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ABSTRACT 

Activity  Vital  Fire  ran  from  19  May  2014  to  30  May  2014.  The  Land  Vehicle  Simulation 
Environment  (LVSE)  was  constructed  to  enable  five  Human  Sciences  experiments  to  be 
carried  out  on  a  single  cohort  of  14  military  personnel.  While  the  activity  was  broadly 
successful,  and  large  amounts  of  quality  data  were  generated,  stability  issues  affected  the 
schedule  during  the  first  week  and  significant  effort  was  required  to  meet  the  requirements  of 
the  researchers.  In  order  to  extract  lessons  learned  from  formal  and  informal  Post  Activity 
Reviews  (PARs)  we  applied  a  hybrid  Thematic  Synthesis  methodology.  The  process  identified 
four  common  themes  in  the  data;  Communication,  Project  Management,  Experimental  Design, 
and  Testing.  Within  these  themes  we  extract  key  observations  and  recommendations. 
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Thematic  Synthesis  of  Post  Activity  Reviews: 
Lessons  Relating  to  Management  of  the  Simulation 
Environment  Supporting  Activity  Vital  Fire  in  May 

2014 


Executive  Summary 


Activity  Vital  Fire  ran  from  19  May  2014  to  30  May  2014.  The  Land  Vehicle  Simulation 
Environment  (LVSE)  was  constructed  to  enable  five  Human  Sciences  experiments  to  be 
carried  out  on  a  single  cohort  of  14  military  personnel.  While  the  activity  was  broadly 
successful,  and  large  amounts  of  quality  data  were  generated,  stability  issues  affected  the 
schedule  during  the  first  week  and  significant  effort  was  required  to  meet  the 
requirements  of  the  researchers.  In  order  to  extract  lessons  learned  from  formal  and 
informal  Post  Activity  Reviews  (PARs)  we  applied  a  hybrid  Thematic  Synthesis 
methodology. 

Each  review  was  decomposed  into  a  list  of  observations.  These  were  sorted  into  an  initial 
list  of  thematic  groups.  To  ensure  completeness,  new  groups  were  created  to  fit 
observations  outside  the  initial  set.  This  larger  set  was  reviewed  to  combine  strongly 
related  themes,  and  then  the  final  list  was  tested  for  balance  by  comparing  the  number  of 
observations  in  each  group. 

The  process  identified  four  common  themes  in  the  data;  Communication,  Project 
Management,  Experimental  Design,  and  Testing.  Within  these  themes  we  extracted  key 
observations  and  the  following  recommendations: 

1.  Clearly  identify  roles  and  responsibilities. 

2.  Disseminate  meeting  minutes  and  key  decisions. 

3.  The  Research  Activity  Information  Development  (RAID)  document  should  be 
reviewed  post-activity  to  ensure  relevance. 

4.  The  RAID  should  be  used  as  a  living  document  to  record  a  history  of  key  decisions. 

5.  Identify  and  adopt  management  best-practice. 

6.  Experiments  should  either  be  designed  to  use  the  existing  and  tested  simulation 
capability  or  submit  requirements  to  be  integrated  into  the  development  program. 

7.  All  experimental  protocols  should  be  piloted  to  confirm: 

o  Equipment  functionality  and  reliability, 
o  Data  collection  and  validation,  and 
o  Contingency  planning. 

8.  To  assist  systematic  post-activity  review,  the  PAR  should  be  formatted  to  address 
the  identified  theme  headings. 
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Acronyms  and  Abbreviations 


BMS 

Battle  Management  System 

LVSE 

Land  Vehicle  Simulation  Environment 

PAR 

Post  Activity  Review 

RAID 

Research  Activity  Information  Development 

UNCLASSIFIED 


UNCLASSIFIED 


DSTO-TN-1408 


1.  Introduction 


Activity  Vital  Fire  ran  from  19  May  2014  to  30  May  2014.  The  Land  Vehicle  Simulation 
Environment  (LVSE)  was  constructed  to  enable  five  Human  Sciences  experiments  to  be 
carried  out  on  a  single  cohort  of  14  military  personnel.  While  the  activity  was  broadly 
successful,  and  large  amounts  of  quality  data  were  generated,  stability  issues  affected  the 
schedule  during  the  first  week  and  significant  effort  was  required  to  meet  the 
requirements  of  the  researchers.  In  order  to  improve  the  capability  it  is  important  to  reflect 
on  the  conduct  of  the  activity  and  to  document  the  lessons  learned. 

2.  Experimental  Requirements 

The  LVSE  was  configured  to  provide  16  motion-actuated  seats  to  simulate  eight  vehicles, 
each  with  a  driver  and  a  co-driver.  Audio  and  video  were  recorded  at  each  station,  and  the 
systems  were  to  be  networked.  The  technical  team  was  responsible  for  managing  the 
construction  of  the  capability,  and  configuring  the  computer  infrastructure  to  support  the 
operation  of  the  experiment.  The  research  team  was  responsible  for  developing  the 
experimental  design,  and  through  negotiation  for  developing  a  set  of  requirements. 
However,  the  research  team  did  not  formalise  their  requirements  until  the  introduction  of 
the  Research  Activity  Information  Development  (RAID)  document.  This  also  required  the 
research  team  to  identify  research  leads  for  each  experiment.  The  final  implementation  of 
the  environment  needed  to  meet  the  specified  requirements  of  each  of  the  five  concurrent 
studies,  described  below. 

2.1  Cognitive  and  metacognitive  performance 

This  study  called  for  the  administering  of  a  battery  of  cognitive  tests  to  be  presented  over 
the  internet  on  tablet  PCs.  The  tablets  were  required  at  designated  times  around  the  clock. 
Data  collection  was  managed  off-site. 

2.2  Impact  of  sleep  loss,  task  demand  and  task  demand  transition  on 
driver,  co-driver  and  crew  performance 

The  support  requirements  of  LVSE  were  to: 

Set  up  and  configure  the  laboratory  hardware  and  equipment  in  such  a  manner  as 
to: 

o  Provide  two-person  'vehicle  crew'  stations  with  driver  and  co-driver 
positions 

o  Allow  customisability  in  configuration,  to  provide  flexibility  in  the 
allocation  of  participants/ crews  to  different  experimental  conditions 
•  Integrate  the  driving  scenario  used  in  a  previous  Caffeine  study  with  the  existing 
experimental  framework  to  provide  an  acceptable  simulation  environment  for  the 
driver 
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•  Provide  equipment  allowing  for  an  acceptable  emulation  of  a  Battle  Management 
System  (BMS)  task  for  the  co-driver 

•  Enable  capture  of  data  relating  to  all  required  measures  of  participant  performance 
and  state  (e.g.  input  devices,  audio  recording,  Optalert) 

•  Store  and  manage  experimental  data  collection  from  the  simulation  environment 

•  Process  and  provide  experimental  data  in  a  format  appropriate  for  analysis  by  the 
research  team 

2.3  Sleep  restriction  and  driver  performance 

•  Time  stamped  audio  from  the  microphones  to  record  sleepiness  ratings  at  5-minute 
intervals. 

•  Driving  performance  data  (speed,  lane  deviations,  and  crashes)  across  the  five 
drives. 

•  Collect  data  from  sleepiness  monitoring  devices. 

2.4  Sleep  restriction  and  co-driver  performance1 

•  Load  the  BMS  software  and  associated  csv.  session  files  onto  a  set  of  tablets 

•  Enable  eight  co-driver  headsets  to  record  audio 

•  Enable  eight  co-driver  headsets  to  receive  live  audio  instructions 

•  Set  up  eight  driver/ co-driver  vehicle  simulation  bays 

•  Load  BMS  files  and  audio  files  onto  shared  drive 


2.5  Convoy  team  study 

The  experiment  was  to  simulate  a  4-vehicle  M113AS4  armoured  platoon  across  a 
minimum  of  seven,  2-hour  scenarios  in  an  immersive  simulation  environment  on  the 
fictitious  island  of  Sahrani.  Each  vehicle  was  to  contain  a  driver  and  a 
commander/ gunner.  The  command  vehicle  for  each  platoon  was  also  to  contain  a  platoon 
commander,  giving  a  total  of  nine  experiment-subjects  in  the  platoon.  The  experiment  was 
to  be  duplicated  across  two  platoons  for  a  total  of  18  subjects.  The  two  platoons  were  to 
operate  in  isolated  VBS2  "worlds",  so  there  would  be  no  interaction.  Although  the 
platoons  were  to  complete  the  same  scenarios  at  the  same  time,  they  would  be  in  separate 
parallel  simulations 

The  convoy  team  study  also  supplied  a  further  91  documented  requirements  in  a 
Functional  Performance  Specification  document.  This  highlights  the  variance  in 
expectations  and  the  level  of  detail  specified. 


1  The  co-driver  is  the  non-driving  crew-member  in  a  two  person  crew.  In  this  case  their  task  was  to  complete 
BMS  interaction  instructions  provided  on  a  tablet  PC. 
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3.  Method  for  extracting  themes 


3.1  Data  Collection 

The  RAID  document  formalised  the  description  of  each  study  and  assisted  the  technical 
team  and  the  researchers  in  confirming  the  requirements  of  each  study  upon  the 
environment.  Following  the  activity  all  research  leads  were  requested  to  complete  a  Post 
Activity  Review  (PAR).  The  PAR  elicited  discrepancies  between  the  support  requested  and 
delivered,  feedback  on  the  support  process  and  on  the  execution  of  the  experiment  during 
the  activity.  This  provided  the  primary  source  of  data  for  lessons  learned.  The  limitations 
inherent  in  this  data  source  are  the  variability  of  the  detail  of  the  responses  and  the 
unstated  assumptions  and  context. 

The  technical  team  also  supplied  a  report  of  the  issues  they  encountered,  areas  they  felt 
needed  improvement  and  the  errors  they  observed  during  the  conduct.  These  reports 
provided  deep  insight,  but  were  informally  structured.  The  author  also  includes  insight 
from  personal  experience,  understanding  of  best  practice  and  unstructured  interviews 
with  stakeholders  and  members  of  both  teams. 

The  data  provide  a  tapestry  of  sometimes  contradictory  insights,  due  in  part  to  context  and 
exposure  to  different  parts  of  the  project  development.  The  data  also  were  captured  after 
the  fact  and  mostly  discussed  the  issues  and  failings  perceived  of  the  completed 
Simulation  Environment  without  taking  into  account  that  the  environment  and  the  team 
developing  it  were  assembled  very  late  in  the  project  plan. 


Figure  1  -A  simulated  land  vehicle  during  the  C2  on  the  Move  component  of  the  activity 

3.2  Data  Analysis 

Simply  collating  the  assessments  can  provide  benefit  to  other  practitioners,  but  only  if  they 
spend  the  time  and  effort  to  assess  which  insights,  observations  and  suggestions  to  act  on 
given  the  variance  of  context  and  conflicting  opinions.  A  systematic  approach  is  needed  to 
provide  a  synthesis  of  the  assessments,  allowing  the  lessons  to  be  translated  to  other 
contexts.  Thomas  &  Harden  (2008)  describe  a  method  to  integrate  the  findings  of  multiple 
qualitative  assessments  called  Thematic  Synthesis.  It  involves  codifying  each  assessment 
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line  by  line,  then  constructing  a  tree  of  analytical  themes.  However,  the  method  relies 
heavily  on  a  single  software  tool  and  a  single  source  of  structured  data. 

Ryan  &  Bernard  (2003)  describe  the  following  12  scrutiny  techniques  to  apply  to  empirical 
data  in  order  to  extract  themes. 


Table  1  -  Scrutiny  Techniques 


1. 

Repetitions 

7. 

Missing  Data 

2. 

Indigenous  Typologies 

8. 

Theory- Related  Material 

3. 

Metaphors 

9. 

Cutting  &  Sorting 

4. 

Transitions 

10. 

Word  Lists  &  Key  Word  Lists  in  Context 

5. 

Similarities  &  Differences 

11. 

Word  Co-occurrence 

6. 

Linguistic  Connectors 

12. 

Metacoding 

Ryan  &  Bernard  (2003)  also  developed  a  method  to  identify  which  techniques  to  apply 
using  the  following  flow-chart. 

Textual  Data? 


Figure  2  -Selecting  among  Theme-identification  Techniques 

In  this  paper  we  have  taken  a  hybrid  approach.  The  procedure  is  taken  from  Thematic 
Synthesis,  and  the  scrutiny  techniques  identified  by  the  above  flow  chart  were  applied  to 
generate  and  validate  the  themes. 
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Most  of  the  data  under  analysis  was  formatted  as  dot-points;  which  are  textual,  but  not 
verbatim  text.  Using  the  flow-chart  we  identify  three  techniques  to  apply:  repetitions, 
similarities  &  differences,  and  cutting  &  sorting. 

Repetition  is  performed  by  searching  for  "topics  that  occur  and  reoccur".  There  is  no 
defined  threshold  of  repetition,  only  subjective  assessment. 

Similarities  and  differences  are  also  called  the  "constant  comparison  method"  and  include 
making  systematic  comparisons  across  units  of  data.  These  units  can  be  line,  by  line, 
random  pairs  of  expression  or  whole  texts. 

Cutting  and  sorting  traditionally  involved  writing  quotes  on  index  cards  and  sorting  these 
into  piles.  In  this  case  we  relied  on  software  rather  than  a  sorting  table. 

1.  Extracting  data  from  studies. 

Each  report  was  broken  down  into  lists.  Most  reports  presented  these  as  a  list  of 
dot  points,  but  continuous  text  was  initially  broken  into  paragraphs  or  sentences 
depending  on  a  subjective  assessment  of  intent.  The  lists  were  then  separated  into 
observations  through  subjective  assessment.  Some  of  these  observations  were 
reworded  to  retain  context. 

2.  Coding  text  and  developing  descriptive  themes. 

Starting  with  an  a  priori  set  of  themes  the  list  of  observations  for  each  report  were 
assessed  to  determine  if  an  observation  and  its  root  cause  could  be  attributed 
somewhere  within  the  set.  If  not,  a  new  theme  was  added.  Report  codifications 
were  compared  to  see  if  common  themes  were  identified  with  contradictory 
observations  or  if  any  theme  was  found  to  be  trivial. 

3.  Context  and  rigour. 

Once  complete  the  set  of  themes  was  reviewed.  Ideally  the  final  set  would  include 
the  key  areas  of  observation  (completeness),  not  allow  for  multiple  codifications 
(orthogonality)  and  contain  a  similar  number  of  observations  (balance).  The  lists  of 
themes  provide  areas  for  discussion,  and  ensure  that  dissenting  observations  are 
highlighted. 


4.  Results 

Using  an  a  priori  list  of  themes2  the  154  observations  were  coded  using  the  method 
described.  At  the  end  of  the  first  pass  the  themes  had  expanded  in  number  to  include: 

•  Communication, 

•  Project  Planning, 

•  Experimental  Design, 

•  Documentation, 

•  Skills, 

•  Capability  Confirmation, 


2  Communication,  Project  Planning,  Experimental  Design,  Data  Validation,  &  Technical  Faults. 
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•  Procedural  Practice, 

•  Flexibility, 

•  Data  Validation, 

•  Contingency  Planning,  and 

•  Technical  Faults. 

These  themes  satisfied  the  criteria  of  completeness  in  that  all  observations  were  coded,  but 
were  neither  orthogonal,  nor  balanced.  Several  observations  relating  to  Project  Planning 
for  instance,  also  related  to  communication  and  management  of  skills.  In  this  case  Project 
Planning  was  renamed  Project  Management  and  observations  falling  outside  its  reduced 
scope  where  reassessed  and  coded  again. 

Finally  the  themes  were  assessed  for  balance.  While  the  number  of  observations  relating  to 
a  theme  does  not  necessarily  correlate  to  its  importance,  it  does  indicate  a  greater 
proportion  of  the  discourse.  Several  of  the  themes  with  smaller  numbers  of  observations 
could  be  considered  repetitions  of  one  of  the  main  points  of  another  theme.  For  instance 
Skills  and  Documentation  each  had  less  than  nine  observations,  but  both  themes  could  be 
considered  repetition  of  a  major  point  under  the  Project  Management  theme. 

Following  this  final  step  of  collation  the  themes  were  identified  as  follows: 


♦  Testing  (38) 


Communication  (24)J© 


Themes 


©•Experimental  Design  (41) 


Project  Management  (51)J© 


Figure  3  - Identified  Themes  (number  of  observations  in  brackets) 

4.1  Communication 

Communication  problems  were  identified  between  most  stakeholders. 

•  Lines  of  communication  were  poorly  defined, 

•  Requirements  were  delivered  late  and  with  insufficient  detail, 

•  Requirements  were  delivered  with  no  room  for  flexibility, 

•  Flexibility  was  specified  as  a  requirement  in  its  own  right, 

•  The  technical  team  failed  to  communicate  their  progress, 

•  Assumptions  of  the  simplicity  of  implementation  were  insufficiently  challenged. 

There  were  also  positive  assessments; 

•  Intent  was  well  understood  and  supported, 

•  The  RAID  was  highly  effective  in  precipitating  clear  roles. 
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4.2  Project  Management 

Many  of  the  failures  in  communication  were  linked  to  the  lack  of  well-defined 
management  structure  and  an  entirely  ad-hoc  management  process.  Staff  management 
and  decision  making  occurred  without  formal  record  or  communication  again,  until  late  in 
the  project. 

•  Lack  of  coordination, 

•  Additional  work  being  taken  on  without  consideration  of  impact. 

Skills  were  identified  as  an  area  of  concern. 

•  Single  points  of  failure  led  to  the  overwork  of  technical  team  members, 

•  Lack  of  project  management  skills  within  the  technical  team  initially, 

•  Team  members  taking  on  tasks  outside  scope  and  expertise. 

A  history  of  poor  documentation  led  to  repetition  of  previous  work. 

•  RAID  introduced  too  late  in  the  process, 

•  No  documentation  of  key  decisions  affecting  design. 

Furthermore,  failure  to  manage  interacting  timelines  impacted  confidence  between  teams 
and  hindered  the  finalisation  of  experimental  design. 

4.3  Experimental  Design 

Experimental  design  suffered  significantly  from  poorly  defined  lines  of  communication. 
Disagreement  on  responsibilities  delayed  the  development  of  requirements,  and  poor 
communication  led  to  the  formation  of  incorrect  assumptions. 

The  tight  timelines  also  removed  much  of  the  opportunity  for  experiment  rehearsal. 

•  Researchers  didn't  fully  appreciate  their  own  requirements  until  after  they  had 
piloted  their  protocols, 

•  Lack  of  practice  left  researchers  with  low  confidence  in  the  running  of  their 
experiments, 

•  Procedural  errors  led  to  data  loss. 

This  theme  highlights  the  interaction  between  all  themes,  poor  project  management 
hinders  communication  leading  to  compressed  timelines,  hindering  successful 
experimental  design  and  reducing  opportunities  for  sufficient  testing. 

4.4  Testing 

Technical  faults  significantly  upset  timelines,  and  their  unpredictable  nature  left  the 
technical  team  exasperated.  Of  key  concern  was  the  failure  to  fully  test  the  capability  in 
advance  as  some  faults  did  not  present  until  the  full  system  was  under  stress. 

•  Audio  devices  failed  when  removing  specific  keyboards. 
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•  User  error,  particularly  with  muting  the  headset,  presented  the  same  perceived 
fault, 

•  Tablets  exhibited  "ghost  touches"  rendering  data  unusable, 

•  There  were  no  contingency  plans  for  most  studies. 

Finally  some  of  the  issues  experienced  impacted  data  collection  and  were  not  uncovered 
until  the  completion  of  the  study. 

•  Sleepiness  assessment  data  could  not  be  processed  by  the  provided  software 
calling  into  question  all  of  that  study's  results, 

•  Data  collection  software  crashed  silently  leaving  gaps  in  the  data. 


5.  Recommendations 


5.1  Communication 

Good  communication  is  essential  for  any  project  and  it  is  essential  that  project 
management  supports  it. 

1.  Clearly  identify  roles  and  responsibilities. 

2.  Disseminate  meeting  minutes  and  key  decisions. 

5.2  Project  management 

The  RAID  document  was  overwhelmingly  supported  for  its  ability  to  formalise 
requirements. 

3.  The  RAID  should  be  reviewed  post-activity  to  ensure  relevance. 

4.  The  RAID  should  be  used  as  a  living  document  to  record  a  history  of  key  decisions. 

The  management  of  staff  was  also  a  concern.  As  on-going  development  is  required  to 
provide  flexibility  to  experimental  design  this  development  should  be  conducted  in  a 
formal  manner,  rather  than  ad-hoc. 

5.  Identify  and  adopt  management  best-practice. 

5.3  Experimental  Design 

6.  Experiments  should  either  be  designed  to  use  the  existing  and  tested  simulation 
capability  or  submit  requirements  to  be  integrated  into  the  development  program. 

5.4  Testing 

Researchers  need  to  proceed  with  confidence  in  the  systems  that  support  them  and  with 
enough  practice  to  be  confident  in  their  own  procedures. 

7.  All  experimental  protocols  should  be  piloted  to  confirm: 

o  Equipment  functionality  and  reliability, 
o  Data  collection  and  validation,  and 
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o  Contingency  planning. 


Finally, 

8.  To  assist  systematic  post-activity  review,  the  PAR  should  be  formatted  to  address  the 
identified  theme  headings. 
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